Surgical navigation with stereovision and associated methods

ABSTRACT

A surgical guidance system has two cameras to provide stereo image stream of a surgical field; and a stereo viewer. The system has a 3D surface extraction module that generates a first 3D model of the surgical field from the stereo image streams; a registration module for co-registering annotating data with the first 3D model; and a stereo image enhancer for graphically overlaying at least part of the annotating data onto the stereo image stream to form an enhanced stereo image stream for display, where the enhanced stereo stream enhances a surgeon&#39;s perception of the surgical field. The registration module has an alignment refiner to adjust registration of the annotating data with the 3D model based upon matching of features within the 3D model and features within the annotating data; and in an embodiment, a deformation modeler to deform the annotating data based upon a determined tissue deformation.

PRIORITY CLAIM & RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/367,243, filed Dec. 2, 2016, which is a continuation-in-part ofInternational Patent Application No. PCT/US15/33672 filed Jun. 2, 2015,which claims priority to U.S. Provisional Patent Application No.62/006,786 filed Jun. 2, 2014, the disclosures of which are incorporatedherein by reference.

International Patent Application No. PCT/US15/33672 is also acontinuation-in-part of U.S. patent application Ser. No. 12/994,044filed on Jan. 21, 2011, now U.S. Pat. No. 9,052,384 issued Jun. 9, 2015,which is a National Stage Entry of PCT Patent Application Serial NumberPCT/US2009/045082 filed May 22, 2009, which claims priority to U.S.Provisional Patent Application No. 61/055,355, filed May 22, 2008, thedisclosure of which are incorporated herein by reference.

International Patent Application No. PCT/US15/33672 is also acontinuation-in-part of International Patent Application No.PCT/US2013/020352 filed Jan. 4, 2013, which claims priority to U.S.Patent Application No. 61/583,092, filed Jan. 4, 2012, the disclosuresof which are incorporated herein by reference.

International Patent Application No. PCT/US15/33672 is also acontinuation-in-part of International Patent Application No.PCT/US2013/024400 filed Feb. 1, 2013, which claims priority to U.S.Provisional Patent Application No. 61/594,862 filed Feb. 3, 2012 andwhich is also a continuation-in-part of PCT Application No.PCT/US2013/020352 filed Jan. 4, 2013, which claims priority to U.S.Patent Application Ser. No. 61/583,092, filed Jan. 4, 2012. Thedisclosures of the above-referenced applications are each incorporatedin their entirety herein by reference.

International Patent Application No. PCT/US15/33672 is related to U.S.patent application Ser. No. 13/145,505, filed in the United StatesPatent and Trademark Office on Jul. 20, 2011, now U.S. Pat. No.8,948,851 issued Feb. 3, 2015, which is a United States National Phaseapplication of International Patent Application No. PCT/US2009/066839filed Dec. 4, 2009, which claims priority to U.S. Provisional PatentApplication No. 61/145,900 filed Jan. 20, 2009. The disclosures of theabove-referenced applications are each incorporated in their entiretyherein by reference.

International Patent Application No. PCT/US15/33672 is also acontinuation-in-part of U.S. application Ser. No. 14/373,443 filed Jul.21, 2014 which is a National Stage Entry of PCT Application No.PCT/US2013/022266 filed Jan. 18, 2013, which claims priority to U.S.Provisional Application Ser. No. 61/588,708, filed Jan. 20, 2012. Thedisclosures of the above-referenced applications are each incorporatedin their entirety herein by reference.

International Patent Application No. PCT/US15/33672 is also acontinuation-in-part of U.S. patent application Ser. No. 14/345,029filed Mar. 14, 2014, which is a national phase of PCT Application No.PCT/US2012/055755 filed Sep. 17, 2012, which in turn claims priority toU.S. Patent Application Ser. No. 61/535,201 filed Sep. 15, 2011. Thedisclosures of the above-referenced applications are each incorporatedin their entirety herein by reference.

GOVERNMENT RIGHTS

This invention was made with government support under grant numbers R01CA159324-01, R01 EB002082-11 and 1R21 NS078607 awarded by the NationalInstitutes of Health. The United States Government has certain rights inthe invention.

SUMMARY

A system for generating a photographic image of a surgical field,includes a stereo image capture device for capturing a stereo imagestream from two cameras imaging the surgical field; and a stereo viewerfor displaying the stereo image stream to a surgeon. The system alsoincludes a stereo image to 3D surface extraction module, implemented asmachine readable instructions stored in memory of a computer, that iscapable of, when the instructions are executed by a processor of thecomputer, generating a first 3D model of the surgical field from thestereo image streams; a registration module, implemented as machinereadable instructions stored in the memory, that is capable of, when theinstructions are executed by the processor, co-registering annotatingdata with the first 3D model; and a stereo image enhancer, implementedas machine readable instructions stored in the memory, that is capableof, when the instructions are executed by the processor, graphicallyoverlaying at least part of the annotating data onto the stereo imagestream to form an enhanced stereo image stream for display by the stereoviewer, wherein the enhanced stereo stream enhances the surgeon'sperception of the surgical field.

In another embodiment, a system configured for graphically representingand displaying a first and second annotating data in relation to asurgical field, including an interactive display device; a registrationmodule, implemented as machine readable instruction stored in a memoryof a computer, that is capable of, when the instructions are executed bya processor of the computer, co-registering the first and secondannotating data; and a stereo image enhancer, implemented as machinereadable instruction stored in the memory, that is capable of, when theinstructions are executed by the processor, graphically overlaying atleast part of the annotating data onto the stereo image stream to forman enhanced stereo image stream for display by the stereo viewer,wherein the enhanced stereo stream enhances the surgeon's perception ofthe surgical field.

In another embodiment, a method for surgical navigation with stereovision, including capturing a stereo image pair of a surgical field;generating a 3D surface model from the stereo image pair; registeringpreoperatively captured annotating data with the 3D surface model;generating an enhanced stereo image based upon the stereo image pair andthe annotating data; wherein the annotating data is graphically overlaidonto the stereo image pair.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows one exemplary system for surgical navigation withstereovision, in an embodiment.

FIG. 2 shows the stereo image enhancer of FIG. 1 in further exemplarydetail.

FIG. 3 shows a portion of the stereo imaging system of FIG. 1 configuredwith stereo hyperspectral cameras for capturing a stereo hyperspectralimage stream, in an embodiment.

FIG. 4 is a flowchart illustrating one exemplary method for surgicalnavigation with stereo vision, in an embodiment.

FIG. 5 shows examples of the annotating data of FIG. 1.

FIG. 6A is a perspective view of one exemplary implantable device modelthat is stored within the system of FIG. 1, in an embodiment.

FIG. 6B is a perspective view of one exemplary implantable device modelthat is stored within the system of FIG. 1 in an embodiment.

FIG. 7 shows one exemplary system with an interactive stereo displaydevice for interacting with a surgeon to analyze and/or create asurgical procedure for a patient, in an embodiment.

FIG. 8 shows the system of FIG. 1 further configured for use inpositioning an implantable device during a surgical procedure, in anembodiment.

FIG. 9 shows one exemplary optical system for surgical navigation withstereovision, in an embodiment.

FIG. 10A is a perspective view of the 3D model of FIG. 1 and theimplantable device model of FIG. 6B.

FIG. 10B is a 2D representation of the enhanced stereo image stream ofthe surgical field corresponding to FIG. 10A as generated by the systemof FIG. 1.

FIG. 11 is a perspective view of the 3D model and the annotating data 3Dmodel generated from the annotating data of FIG. 1, where the annotatingdata is based upon MRI tomography data.

DETAILED DESCRIPTION OF THE EMBODIMENTS

FIG. 1 shows one exemplary system 100 for surgical navigation withstereovision. System 100 includes a stereo viewer 102, a stereo imageannotator or enhancer 104, and a stereo image capture device 106. System100 is used to aid a surgeon during an operation on a person 180. In theexample of FIG. 1, the surgeon is performing a procedure on a brain 182of person 180. However, system 100 may be used to aid procedures onother parts of a person's body, and may be used when performingprocedures on animals as well as on humans.

Stereo image capture device 106 includes illumination sources 114, 116and at least two cameras 110, 112, that cooperate to capture a highquality stereo image stream 107 of a surgical field 184 (i.e., the areawhere the procedure is performed). System 100 may be calibrated usingone or more calibration targets used in surgical field 184, as taught byPCT/US2009/045082.

Stereo image enhancer 104 receives stereo image stream 107 from capturedevice 106 and generates an enhanced stereo image stream 103 from stereoimage stream 107 by adding information from annotating data 108.Annotating data 108 is for example a previously stored imaging data setsuch as one of (a) preoperative, intraoperative and postoperativeradiological studies selected from the group including MRI, fMRI, CT,SPECT, and PET, and (b) physiological studies selected from the groupincluding EEG, evoked potentials, and magnetoencephalography (MEG).

At least three reference points 118, 120, 122 may be attached to patient180 for the duration of the procedure and the location of each referencepoint is defined as a location within a patent-centered coordinatesystem of patient 180. A tracker 129 determines, relative to itself,locations of reference points 118, 120, 122 attached to patient 180 andreference points 124, 126, 128 attached to stereo image capture device106. Registration module 132 uses these locations to determine a viewreference 131 (e.g., spatial relationship including relative positionand orientation) between patient 180 and stereo image capture device106. View reference 131 defines a spatial relationship of stereo imagestream 107 relative to surgical field 184, and may also incorporatefunctions (e.g., zoom) of optics within cameras 110 and 112. Tracker 129is in some embodiments a commercially available operating room trackingdevice that determines positions of reference points or transpondersattached to equipment in the operating room, including stereo imagecapture device 106, and may include transponders attached to thepatient. Tracker 129 may utilize one or both of optical andelectromagnetic tracking as known in the art. In an alternateembodiment, where stereo image stream 107 includes images of referencepoints 118, 120, and 122, registration module 132 may automaticallydetermine a spatial relationship between stereo image capture device 106and patient 180, or to a frame attached to patent 180. 3D model 134allows each pixel within stereo image stream 107 to be located withinthe patient-centered coordinate system. By determining this relationshipbetween pixels of stereo image stream 107 and the patient-centeredreference system, image post-processing techniques may be used toevaluate captured images.

In a particular embodiment where registration module 132 automaticallydetermines spatial relationship between stereo image capture device andpatient 180, the registration module determines features of patient bonestructure from preoperative CT scans and/or MRI scans, and registersthis bone structure to features of bone structure determined from otherCT scans and MRI scans for the same patient as well as to bone structuredetermined from 3D surfaces extracted from the stereo images. In theevent an inconsistency, such as a scan with significant bony differencesfrom other scans associated with the patient, or features of facial 3Dsurfaces that are inconsistent with features of bone structure extractedfrom the 3D surfaces, is found while automatically registering scans, awarning message is generated and provided to the surgeon.

It should be noted that stereo image stream 107 is recorded throughout asurgical procedure, and the recorded stereo image stream is effectivelycontinuously geotagged by attaching an encoded position and orientationof image capture device 106 as determined by tracker 129 in real-time.The recorded image stream is available for documentation, and imagepairs from the image stream are sampled and processed as indicatedherein for extraction of 3D surface models and for co-registration of,and co-presentation of, data from other image sources, as describedherein, periodically throughout a surgical procedure.

Stereo image enhancer 104 may be controlled, in a particular embodimentby the surgeon or a surgical assistant, to graphically enhance stereoimage stream 107 with selected information from annotating data 108 suchthat the surgeon has an enhanced view of surgical field 184 throughstereo viewer 102. Enhancer 104 includes a stereo image to 3D surfacemodule 130 that generates a 3D model 134 of surgical field 184 fromstereo image stream 107. 3D model 134 includes a reference 135 thatdefines a spatial relationship of 3D model 134 relative to thepatient-centered coordinate system of patient 180, in a particularembodiment the spatial relationship is determined from tracker 129 basedupon reference points 118, 120, 122. Although the patient-centeredcoordinate system described herein, other coordinate system may be usedwithout departing from the scope hereof. For example, an operating-roomcentered coordinate system may be used, wherein the determined spatialrelationship between patient 180 and stereo capture device 106 allowsconversion of coordinates from one coordinate system to anotheraccording to a calibration.

Registration module 132 operates to co-register annotating data 108(based upon a reference 109 associated, or included with, annotatingdata 108) and 3D model 134 (based upon determined reference 135) andgenerate graphical information from annotating data 108 that correspondsto surgical field 184 as imaged within image stream 107. For example,where annotating data 108 is a tomographic image generated from a MRIscan, that image may be scaled and oriented with respect to the surgicalfield viewed by the surgeon through stereo viewer 102 and graphicallyadded to enhanced stereo image stream 103. Enhancer 104 thereby enhancesthe view of surgical field 184 seen by the surgeon by adding informationfrom other sources and co-registering this information to a conventionaloptical view; thus the surgeon assimilates the information without theneed to move or avert his/her eyes from the surgical field. In anotherembodiment, the surgeon may view a corresponding view of 3D model 134and a 3D model of annotating data 108, and may view data from the 3Dmodel 134 with superimposed annotating data 108 as rendered stereoimages, as wireframe models, as one or more tomographic slices, asfalse-color renderings embodying color-encoded depth, as a combinationof these representations.

Applicant notes that not only may the coordinate systems used torepresent images from each modality may differ—for example an MRI mayuse a rectilinear coordinate system while an ultrasound imaging modalitymay use a polar coordinate system, but each series of images within amodality may be offset relative to other images of that modality=forexample two successive MRIs may be offset relative to each other. Amapping between raw image coordinates and patient coordinates isestablished and, and images transformed from raw coordinate systems intothe patient centered coordinate system, by registration module 132 foreach image.

Registration module 132 may identify features within stereo image stream107 and/or 3D model 134 that correspond to features identified withinannotating data 108, wherein registration module 130 automaticallyaligns graphical images generated from annotating data 108 with stereoimage stream 107. This automatic alignment may include one or more ofshifting, rotating, warping, and scaling of annotating data 108 to aligncorresponding features between annotating data 108 and stereo imagestream 107. Registration module 132 may also identify tissue shiftwithin 3D model 134 as compared to annotating data 108, whereinregistration module 132 may shift, rotate, warp, and/or scale annotatingdata 108 to correspond to 3D model 134. Brain, and many other organs,are soft, deformable, tissues. For example, when prepared for surgery,soft tissue may move as a result of skull removal, retraction, incision,and the like. Similarly, during surgery, tissue and inclusions thereinsuch as tumors or critical structures may shift position. Stereo imageto 3D surface module 130 continually updates 3D model 134 based uponstereo image stream 107 and registration module 132 continually compares3D module 134 to annotating data 108, shifting, rotating, warping,and/or scaling annotating data 108 to match 3D model 134. In oneembodiment, registration module 132 utilizes a mechanical model oftissue, instruments, and structure within surgical field 184 such thataccurate manipulation of annotating data 108 may be made correspondingto detected movement within 3D model 134.

In one embodiment, where enhanced data 108 does not include imagery(i.e., EEG and/or evoked potentials), enhancer 104 generates contourlines mapped to 3D model 134 and corresponded to information withinenhanced data 108.

In another embodiment, where enhanced data 108 includes a 3D model ordata derived in a voxel-based form (e.g., MRI data), enhancer 104 maygenerate one or more cross sections through an indicated incision planeand map that cross section to the incision plane within 3D model 134such that the surgeon views the corresponding 3D structures beneath theincision plane prior to, and after, making the actual incision. Bydisplaying structure that is not readily or directly visible, thesurgeon is made aware of such structures and their relative positionswithout the need to look away from the surgical field to reference othersources that may distract the surgeon from the procedure, or which maynot represent current positions of the structures. That is, system 100makes information of enhanced data 108 available within the same fieldof view as surgical field 184, thereby lessening need for the surgeon tolook away from stereo viewer 102 or from surgical field 184.

System 100 includes an interactive controller 170 adapted to select oneof a plurality of predefined stored views 160 of 3D model 134 andannotating data 108. In one embodiment, controller 170 is controlled bya surgeon's foot. In another embodiment, controller 170 is controlled bythe surgeon's eye movement, wherein controller 170 may be implementedwithin stereo viewer 102. In yet another embodiment, controller 170 iscontrolled by a surgeon's voice using a trained speech interpretationinterface. Stored views 160 define a plurality of views that combineannotating data 108 with stereo image stream 107 to generate enhancedstereo image stream 103. For example, one view may select a particularannotating data 108 for enhancing stereo image stream 103. Each viewwithin stored views 160 also defines which portions of annotating data108 to select for combining with stereo image stream 107. For example,where a surgical incision is planned (see planned surgical procedures760 of FIG. 7), stored view 160 may define a view that graphicallyindicates planned incisions within enhanced stereo image stream 103, andmay further define a corresponding portion of annotating data 3D model222 for display with the planned incision to indicate tissue structuresbeneath the intended point of incision. This provides the surgeon withadditional information relative to the next immediate step in theplanned operational procedure.

Stored views 160 may also define an alternate viewing point for displaywithin stereo viewer 102, wherein the stereo image enhancer 104generates the alternate view based upon 3D model 134 and annotating data108. For example, a surgeon may elect to temporarily view a side imageof a patient's head to learn additional information from MRI tomographydata stored within annotating data 108 in relation to surgical field184.

FIG. 2 shows stereo image enhancer 104 of FIG. 1 in further exemplarydetail. Registration module 132 includes an alignment refiner 202 thatutilizes a shifting/rotating/scaling/warping module 204 to improveregistration of annotating data 108 with 3D model 134. Registrationmodule 132 may also include a deformation modeler 206 that generates a3D mechanical model 208 of annotating data 108 to model deformation oftissue within annotating data 108 based upon one or more of gravity,retraction, incision, and so on. Deformation modeler 206 therebydetermines an appropriate deformation of annotating data 108 to matchdetermined deformation of tissue within 3D model 134 and surgical field184. In one example, a portion of a patient's skull is removed forsurgery, resulting in reduced pressure and support for soft braintissues. This reduced support allows the brain tissue to reposition as aresult of gravity and reduced support. Deformation modeler 206 uses 3Dmechanical model 208 to estimate such deformation and alignment refiner202 uses shifting/rotating/scaling/warping module 204 to apply theestimated deformation to annotating data 108 such that annotating data108 better aligns to 3D model 134. Similarly, when a surgeon appliesretraction, or makes an incision, deformation modeler 206 uses 3Dmechanical model 208 to estimate deformation of surrounding tissue tobetter align annotating data 108 to 3D model 134.

In one embodiment, 3D mechanical model 208 includes learning capabilityand is continually updated based upon detected deformation of actualtissue during the surgery. In an embodiment, stereo pairs captured usingthe stereo image capture device 106 are processed to extract a 3Dsurface model, which in turn is used to determine an actual surfaceconfiguration that is fed back to mechanical model 208; in a particularembodiment these 3D surface models are used to tune coefficients of aneural network implementation of a mechanical model 208. For example,each time 3D model 134 is updated based upon a performed surgicalprocedure, a mechanical property extraction routine 231 updates 3Dmechanical model 208 to better predict the outcome of the procedure.

Stereo image enhancer 104 may also include an annotating data 3D modelgenerator 220 that generates an annotating data 3D model 222 fromannotating data 108. For example, where annotating data 108 representsan MRI scan, annotating data 3D model 222 is generated to represent amodel of the scanned tissue. Where annotating data 108 represents datafrom a physiological study (i.e., physiological data) selected from thegroup including EEG, evoked potentials, and magnetoencephalography(MEG), annotating data 3D model generator 220 may generate annotatingdata 3D model 222 using contours based upon 3D model 134 or known shapeof the patient.

Electrical activity may be recorded by EEG electrodes at particulartimes during a preoperative or an intraoperative EEG study, and mayinclude recorded evoked potentials. Preoperative studies may useconventional scalp electrodes, while intraoperative studies may use anycombination of scalp, cortical surface, and deep electrodes appropriateto the surgical procedure and stage of surgery at which EEG study isdesired. This electrical activity is mapped onto a 3D model, for exampleas extracted from a CT scan or MRI imaging study, of the subject'sbrain. The resulting 3D maps of electrical activity are processed byannotating data 3D model generator 220 to generate annotating data 3Dmodel 222 that is then co-registered with 3D model 134 extracted fromstereo image stream 107. In this way, EEG activity associated with anepileptic focus may be used to enhance the surgeon's view of thesurgical field. EEG activity associated with evoked potentials, such aspotentials evoked when the subject speaks, may thus be added to thesurgeon's view of the surgical field. For example, EEG activityassociated with particular areas, such as Broca's area critical forspeech, or visual cortex associated with the fovea, may be incorporatedwithin enhanced stereo image stream 103.

Mechanical property extraction routine 231 updates 3D mechanical model208 based upon observed displacement of tissue, such as when an indenteris place on the brain for example.

Stereo image enhancer 104 may also include a physiological tissueproperty analyzer 230 that processes intraoperative visual informationfor determination of tissue physiological properties, including but notlimited to blood flow and oxygenation (oxygenation is determined bymeasuring tissue absorbance parameters at each of several wavelengths oflight), light scattering and absorbance parameters, fluorescence ofphysiologic biomarkers, various chromophores including heme andporphyrins, and fluorescent optical biomarkers such as may be producedby metabolism of prodrugs. System 100 may have an integral tissueclassifier based upon these physiological properties. For example,analyzer 230 may map and display tissue classifications and/or thepresence or absence of particular degrees of oxygenation, lightscattering, and absorbance parameters. Such physiological properties maybe indicated within enhanced stereo image stream 103 and/or outputseparately. Since tumor cells may have different average sizes thannormal cells, and many tumors are hypoxic, this information may be usedto locate tumors.

Further, since patients may be conscious during surgery, blood flow andoxygenation is dependent upon neurological activity, and oxygenation ismeasured by the hyperspectral portion of the system by observingdifferences in absorption due to the spectral shift of hemoglobin withoxygenation, the system may perform intraoperative functionalneuroimaging. Intraoperative functional neuroimaging is of use inisolating structures that require preservation, or in locating epilepticfoci.

Registration module 132 may first align each model 134, 208, 222 andannotating data 108 based upon references 135, 209, 223, and 109 thatdefines a relationship of the model/data to a patient-centered referencesystem for example, and thereby to one another. Registration module 132may then use alignment refiner 202 to better register two or more modelstogether, such as by deforming annotating data 108 to align identifiablefeatures within annotating data 108 with corresponding features within3D model 134.

An image generator 250 then generates enhanced stereo image stream 103based upon stereo image stream 107 and rendering of one or more of 3Dmodel 134, 3D mechanical model 208, annotating data 3D model 222, andannotating data 108. For example, at least part of one or more of 3Dmodel 134, 3D mechanical model 208, annotating data 3D model 222 andannotating data 108 may be graphically rendered and overlaid onto stereoimage stream 107 to form enhanced stereo image stream 103.

Fluorescence/Hyperspectral Enhancement

FIG. 3 shows a portion of stereo imaging system 100 of FIG. 1 configuredwith stereo hyperspectral cameras 310, 312 for capturing a stereohyperspectral image stream 307. In a first embodiment, each camera 310,312 has a monochrome sensor and a controllable color filter 311, 313,wherein the filter is controlled to selectively capture images of aparticular color band. In a second embodiment, each camera 310, 312 hasa high resolution sensor that is fitted with pattered filters having aplurality (e.g., 64) of color bands, wherein the camera simultaneouslycaptures images in each of the color bands. These cameras operatesimilarly to conventional color imagers that typically have Bayerfilters for separating the captured image into three color bands (red,green, and blue). However, in this embodiment, the image is separatedinto 64 (or more) distinct color bands (at the expense of resolution).In a third embodiment, each camera 310, 312 has a scanning slit with adispersing element (e.g., prism or grating) that disperses the imagefrom each point along the slit across multiple pixels of a sensor array,and thereby provides a spectrum, including measurements of intensity atas many as 1024 wavelengths, at each point along the slit. An opticalsystem is configured such that the slit traverses a field of view, and ahyperspectral cube is captured of the entire field of view.

In one embodiment, cameras 310, 312 replace camera 110, 112, whereinprocessing within stereo image capture device 306 recombines stereohyperspectral image steam 307 to form stereo image stream 107. In thisembodiment, since stereo hyperspectral image stream 307 and stereo imagestream 107 are generated from the same captured images, co-registrationis ensured.

In another embodiment, cameras 310, 312 are included with cameras 110,112 such that image streams 107 and 307 are generated independently.

Within enhancer 104, a hyperspectral image processor 302 receives stereohyperspectral image stream 307 and generates a 3D hyperspectral model304 with a reference 305. Fluorescence depth processor 310 operates togenerate a 3D fluorescence model 312 with a reference 313 fromhyperspectral stream 307; in an embodiment fluorescence depth processor310 determines depth in tissue of fluorophores of 3D fluorescence model312 by determining observed fluorescent emissions spectra—the intensityof observed fluorescent emissions in hyperspectral stream 307 at two ormore wavelengths. The depth processor 310 models spectral shift due toabsorption and scattering of light by tissue at those two or morewavelengths relative to known emissions spectra of the fluorophores.Since each fluorophore emits light with a known emissions spectra ordistribution of intensity of emitted light at the two or morewavelengths, and tissue absorbs and scatters light at each wavelengthdifferently, deeper fluorophores generally have a greater spectral shiftbetween observed fluorescent emissions spectra and the known emissionsspectra of the fluorophores.

Registration module 132 operates to co-register two or more models 134,208, 222, 304, and 312, and image generator 205 operates to graphicallycombine at least part of one or more models 134, 208, 222, 304, and 312with stereo image stream 107 to form enhanced stereo image stream 103.In an embodiment, hyperspectral image processor 302 also is adapted tocapture a first fluorescent image and a second fluorescent image atdifferent wavelengths and to produce a third or difference imageaccording to a formula (third image)=K1 (K2*(first image)−(secondimage)) for some real numbers K1 and K2. In an alternative embodiment, adecay rate image is generated by fitting intensity I of pixels of asequence of N fluorescent images indexed by integer k taken over aperiod of time T are fitted to an exponential decay functionI(k)=I(1)10^(kTd) where I(1) is an initial intensity after a fluorescentagent is administered, and d is a per-pixel decay rate;

Captured hyperspectral/fluorescent image streams, and/or determined 3Dmodels therefrom, may be stored for use as annotating data 108 in afollowing procedure/operation. For example, a first part of a surgicalprocedure may collect hyperspectral/fluorescence data from a patient,wherein that data is stored within system 100 and used (e.g., asannotating data 108) within a subsequent part of the procedure.

FIG. 4 is a flowchart illustrating one exemplary method 400 for surgicalnavigation with stereo vision, in an embodiment. In step 402, method 400captures a stereo image stream of a surgical field. In one example ofstep 402, stereo image capture device 106 captures stereo image stream107 of surgical field 184. In step 404, method 400 generates a 3Dsurface model from the stereo image stream. In one example of step 404,stereo image to 3D surface module 130 generates 3D model 134 based uponstereo image stream 107. In step 406, method 400 generates a 3Denhancing model from annotating data. In one example of step 406,annotating data 3D model generator 220 generates annotating data 3Dmodel 222 from annotating data 108.

In step 408, method 400 registers the 3D enhancing model with the 3Dsurface model. In one example of step 408, registration module 132aligns annotating data 3D model 222 with 3D model 134 based uponreference 223 of model 222 and reference 135 of model 134. In step 410,method 400 adjusts registration of the 3D enhancing model based uponmatched features between the 3D enhancing model and the 3D surfacemodel; in an embodiment features are matched between these 3D models bymanual tagging, in an alternate embodiment common features of the modelsare matched automatically. In one example of step 410, alignment refiner202 is invoked by registration model 132 to better register/alignidentifiable features within annotating data 108 to correspondingfeatures within 3D model 134. In another example of step 410, alignmentrefiner 202 invokes deformation modeler 206 to annotate a 3D mechanicalmodel 208 with annotating data 108 and then deform 3D mechanical model208 based upon alignment of corresponding identified features of 3Dmodel 134 to determine post-deformation locations of features found inannotating data 108. Alignment refiner 202 then invokesshifting/rotating/scaling/warping module 204 to apply the identifiedchanges of 3D mechanical model 208 to annotating data 3D model 222.

In step 412, method 400 generates an enhanced stereo image stream basedupon the stereo image stream and the annotating data 3D model. In oneexample of step 412, image generator 250 generates enhanced image stream103 based upon stereo image stream 107 and at least part of enhanceddata 3D model 222.

In step 414, method 400 outputs the enhanced stereo image stream. In oneexample of step 414, image generator 250 outputs enhanced stereo imagestream 103 to stereo viewer 102.

FIG. 5 shows examples of annotating data 108 of FIG. 1. As noted above,annotating data 108 is a previously created set of data from one ofradiological studies and physiological studies of the patient. Each dataset may be independently modeled by generator 220 to generate anindependent annotating data 3D model 222 that may be included at leastin part within enhanced stereo image stream 103.

In addition to live hyperspectral, fluorescence depth-resolved,differential fluorescence, and other images, annotating data 108 mayinclude one or more of: a recorded hyperspectral image stream 502 thathas an associated reference 503, a recorded fluorescence image stream504 that has an associated reference 505, a recorded fluorescencedifference image stream, a recorded hyperspectral image model 506 thathas an associated reference 507, a recorded fluorescence image model 508that has an associated reference 509, a recorded stereo image stream 510that has an associated reference 511, a recorded 3D model 512 that hasan associated reference 513, MRI data 514 that has an associatedreference 515, an fMRI image 516 that has an associated reference 517, aCT image 518 that has an associated reference 519, a single-photonemission computed tomography (SPECT) image 520 that has an associatedreference 521, a PET image 522 that has an associated reference 523, anEEG data set 524 that has an associated reference 525, an evokedpotentials data set 526 that has an associated reference 527, and a MEG(Magnetoencephalographic) data set 528 that has an associated reference529. For each data set or image, the associated reference relates thedata to the patient-centered coordinate system such that system 100 mayrelate one set of data with another, and each set of data with 3D model134.

FIG. 6A shows a perspective view of one exemplary implantable devicemodel 600 that is stored within system 100. Model 600 represents anelectrode array that may be implanted subdural, where positioning of thedevice during surgery is critical to successful operation.

FIG. 6B shows a perspective view of a model 650 of one exemplaryimplantable device model 650 that is stored within system 100. Model 650represents a deep brain probe, deep brain stimulator, electrode array,or other device, that should be inserted into a brain such that a tip652 of the probe is positioned within a target area of the brain.Positioning of the device during surgery is critical to successfuloperation and is further complicated in that a surgeon is unable tovisually see the location of tip 652 with respect to the target area. Acurrent position of the implantable device is determined bytriangulation in the 3D images taken by the 3D imaging device usingreference marks on the rear or top of the implantable device and knownpositions of the 3D imaging device.

In an embodiment, during surgery, and in order to facilitate accurateplacement of implantable devices, probes, or similar tools, a current 3Dmodel of tissue is rendered and displayed. The current 3D model isdisplayable from either the point of view of a tip of the implantabledevice, or from a point of view of a surgeon with a representation ofthe implantable device, including a representation of portions of thedevice that are hidden from view by tissue. The system provides forsuperimposition of modeled locations, as displaced during surgery, offeatures extracted from annotating data, including a target area for theprobe tip in the brain. A surgeon viewing the target area from the pointof view of the probe tip can aim the probe at the target area, and byviewing the probe and target area as viewed from a different angle, canjudge how much further the probe should be inserted to reach the targetarea.

FIG. 7 shows one exemplary system 700 with an interactive stereo displaydevice 701 (e.g., stereo viewer 102 of FIG. 1, a 3D television typedisplay (for use with 3D glasses), touch screen, keyboard and mouse, andso on) for interacting with a surgeon to analyze and/or create asurgical procedure for a patient. In some embodiments, system 700 isused outside of an operating theater to plan a surgical procedure using3D models of the patient. Recorded data 190 from a previous surgeryand/or annotating data 108 captured during a previous procedure on thepatient are co-registered and presented for interaction with thesurgeon. System 700 has a computer 702 with a processor 704 that iscommunicatively coupled with a memory 706. Memory 706 is shown storing sregistration module 732 and an image generator 750. Registration module732 and image generator 750 have similar functionality to registrationmodule 132 and image generator 250 of system 100, FIGS. 1 and 2.Registration module 732 and image generator 750 are at least partlyimplemented as machine readable instructions that are loaded andexecuted by processor 704 to perform the functionality described herein.Registration module 732 co-registers two or more models to present acoherent stereovision display, that, if desired by the surgeon, showsfeatures of derived from multiple preoperative imaging modalities suchas MRI, CT, and ultrasound imaging to a surgeon using system 700.

The surgeon interacts with interactive 3D display device 701 tomanipulate models of one or more of an implantable device, defineplanned surgical procedures 760, and define views within stored views160 for use during a procedure. For example, system 700 allows a surgeonto plan an operation by manipulating views (e.g., rendered images thatmay include appropriate shading and simulated lighting) of one or more3D models 734, 600, 722, defining the planned surgical procedures 760,such as making incisions, positioning an implantable device, andpreselecting the appropriate views (stored within stored views 160) ofannotating data 108 to support the operation.

In one example of operation, the surgeon positions a model of animplantable device relative to a 3D model of a patient's brain generatedfrom MRI data. The surgeon defines one or more views of the modeledimplant and modelled MRI data (and any other annotating data 108 thatmay provide useful information) that will provide additional guidance tothe surgeon during the operation. Implantable device model 600, plannedsurgical procedures 760, and stored views 160 are then used withinsystem 100 to provide additional support to the surgeon during theoperation.

In another example of operation, the surgeon selects previously recordeddata 190 and annotating data 108 for a patient and operates system 700to generate a view of at least a portion of the patient based upon anindicated surgical field (e.g., surgical field 184, FIG. 1). The surgeonmay adjust the field of view to synchronously rotate the viewed image.Thus, the surgeon may select a stereo view of the models that is optimalfor positioning an implantable device, or for planning a surgicalprocedure. Registration module 732 aligns (co-registers) a 3D model 734generated from recorded data 190 with an annotating data 3D model 722generated from annotating data 108 such that the generated view of 3Dmodel 734 and annotating data 3D model 722 are consistent. For example,where annotating data 3D model 722 is based upon MRI tomographic images,the surgeon may select a particular cross section or portion ofannotating data 3D model 722 for display within stereo viewer 102 duringthe operation.

As with system 100, registration module 732 aligns 3D model 734 andannotating data 3D model 722 based upon reference 735 and reference 723,respectively. Registration module 732 may also include an alignmentrefiner (similar to alignment refiner 202, FIG. 2) to ensure correctalignment of 3D model 722 with 3D model 734.

Planned surgical procedures 760 define surgical activities that thesurgeon intends to perform on the patient. These activities are selectedfrom the group including resection, retraction, trajectorydetermination, and surgical corridor, and may be defined singly or incombination. These surgical activities are thus defined preoperativelywith respect to one or more imaging data sets previously captured of thepatient.

System 700 may include a 3D mechanical model 708 (similar to 3Dmechanical model 208 of FIG. 2) that is used to model tissue in at leastpart of the surgical field and/or annotating data 3D model 722 such thattissue displacement resulting from planned surgical procedures 760 arepredicted and portrayed correctly.

FIG. 8 shows system 100 of FIG. 1 further configured for use inpositioning an implantable device during a surgical procedure based uponimplantable device model 600 and associated reference 601, wherereference 601 is defined by system 700 of FIG. 7 as the intendedposition of the implantable device within the patient.

As previously described, system 100 invokes stereo image to 3D surfacemodule 130 to generate 3D model 134 based upon stereo image stream 107.Registration module 132 then aligns (co-registers) annotating data 3Dmodel 122 and implantable device model 600 with 3D model 134 based uponreferences 123, 601, and 135. System 100 may then refine this alignmentusing alignment refiner 202 (not shown in FIG. 8) as described above.

System 100 may also include an implant recognition module 802 thatoperates to detect the location of an implant within 3D image stream 107based upon implanted device model 600. Implant recognition module 802may determine position and orientation of the implant even when only apart of the implant is visible within stereo image stream 107. In oneembodiment, the implant includes identification and orientation markingsthat aid recognition and orientation by implant recognition module 802.

By determining position and orientation of the implantable device andthe desired position of the implantable device, system 100 may generategraphical guides to aid the surgeon's positioning and insertion of thedevice. In particular, by modeling deformation of tissues (includingtarget and other non-visible structures) resulting from applied pressureand/or displacement, system 100 provides guidance for more accuratepositioning of the implantable devices as compared to using conventionalplacement means. Further, where the implantable device passes adjacentcritical structures within the tissue, system 100 may provide warnings(visual and/or audio) to the surgeon where the device is approachingsuch critical areas. Visual indications may include 3D graphical displayof target and critical structures and a 3d graphical representation ofthe implantable device as it moves through the tissue during theprocedure.

As shown in FIG. 1, system 100 may record one or both of stereo imagestream 107 and enhanced stereo image steam 107 throughout the surgicalprocedure/operation. System 700 of FIG. 7 may they be used to perform apost-operative analysis of the surgical procedure based upon therecorded information and annotating data 108, wherein system 700automatically refines alignment and uses 3D mechanical model 702 toallow for tissue deformation resulting from activities during theoperational procedure. Recorded data 190 thereby provide documentationof the surgical procedure for future reference and analysis alone orco-registered with other imaging data sets of annotating data 108, evenwhere the annotating data 108 was not used during the surgery—such aswhen obtained post-surgery.

Recorded data 190 and annotating data 108 may be used for postoperativeuse of intraoperatively acquired functional information (including butnot limited to motor, sensory, speech, memory, and other cognitivefunction). For example, recorded data 190 may be co-registered withother annotating data 108 acquired postoperatively.

Recorded data 190 (e.g., hyperspectral stereo imaging data) andannotating data 108 may be used to determine abnormal tissuephysiological properties, including but limited to blood flow and tissueoxygenation, and to concentrations of selective and nonselectivefluorescent optical biomarkers.

Recorded data 190 may be analyzed using system 700 together withsubsequently acquired and potentially co-registered imaging data sets,either extraoperatively or intraoperatively at any subsequent surgery.

FIG. 10A is a perspective view of 3D model 134 and implantable devicemodel 650 illustrating a tip 1000 of a partially positioned devicecorresponding to model 650 in relation to a target area 1004 withinbrain 182. FIG. 10B is a 2D representation of enhanced stereo imagestream 103 (i.e., the surgeon's view) of surgical field 184corresponding to FIG. 10A. As appreciated, the 3D view resulting fromenhanced stereo image stream 103 provides significantly more informationthat can be shown in the 2D example of FIG. 10B. FIGS. 10A and 10B arebest viewed together with the following description.

The position and orientation of the device being implanted is detectedby implant recognition module 802 and model 650 positioned relative to3D model 134 accordingly. Target area 1004 is defined within plannedsurgical procedures 760 for example, and may include a desired entrypoint 1005 and/or path for the implantable device. A view of models 134,650 and target area 1004 is generated corresponding to the surgeon'sview of surgical area 184 and added to enhanced stereo image stream 103such that the surgeon sees, in 3D, a representation 1052 (e.g., arendered image) of tip 652 and target area 1004 in relation to surgicalfield 184. The surgeon may also view models 134, 650 and target area1004 from other perspectives during the activity of inserting theimplantable device. Since the position of the device being implanted isdetermined from stereo image stream 107 and is accurately co-registered(aligned) with the tissue of patient 180 based upon 3d model 134, thedisplayed alternate view is also accurate and updated in real-time.Implantable device 1002 may have one or more visibly detectable markingsto facilitate determining orientation of device 1002 by implantrecognition module 802.

Other annotating data 108 may also be selected for display to thesurgeon. FIG. 11 is a perspective view of 3D model 134 and annotatingdata 3D model 222 generated from annotating data 108 based upon MRItomography data. For clarity of illustration, model 650 and target area1004 are not shown in the example of FIG. 11, but could be selectivelyadded by the surgeon to show progress of tip 652 towards target area1004.

In the example of FIG. 11, a section of annotating data 3D model 222 issuperimposed onto a cut-away of 3D model 134 to emphasize the innerstructures of tissue below the surface of 3D model 134. System 100generates enhanced stereo image stream 103 to superimpose the hiddentissue structure onto the surgeon's view of surgical field 184. Sincemodel 222 is aligned with 3d model 134 and corrected based upon 3Dmechanical model 208, system 100 enhances the surgeons view with a veryaccurate 3D portrayal of the tissue normally hidden from the surgeon'sview.

FIG. 9 shows one exemplary system 900 for surgical navigation withstereovision. System 900 includes an optical surgical microscope 902 anda stereo image enhancer 904 that receives a stereo image stream 907 of asurgical field 984 from cameras 910 and 912. System 900 operatessimilarly to system 100 of FIG. 1 to enhance a surgeon's view of asurgical field 984.

At least three reference points 918, 920, 922 may be attached to patient980 for the duration of the procedure and the location of each referencepoint is defined as a location within a patent-centered coordinatesystem of patient 980. As with system 100, tracker 129 determineslocations of reference points 918, 920, 922 attached to patient 980 andreference points 924, 926, 928 attached to microscope 902. Stereo imageenhancer 904 uses these locations to determine a view reference (e.g.,spatial relationship including relative position and orientation)between patient 980 and microscope 902 to define a reference of stereoimage stream 907 captured by cameras 910, 912. Stereo image enhancer 904generates and sends an enhancing image stream 903 to image projectors914, 916 that project the images from stereo image stream 903 intooptical paths 932, 934, respectively. As shown, optical path 932includes a first beam splitter 942 that diverts a portion of light fromsurgical field 984 to camera 910 and a beam combiner 946 that adds lightfrom image projector 914 to optical path 932. Similarly, optical path934 includes a first beam splitter 944 that diverts a portion of lightfrom surgical field 984 to camera 912 and a beam combiner 948 that addslight from image projector 916 to optical path 934.

Combinations

The features and capabilities herein described may be combined in manydifferent ways into a system; for example but not limitation thefluorescent depth processor and fluorescent stimulus light sources maybe in an optional, extra-cost, module found in some systems but not inothers. Among other combinations of features anticipated include:

A system designated A for generating a photographic image of a surgicalfield, including a stereo image capture device for capturing a stereoimage stream from two cameras imaging the surgical field; and a stereoviewer for displaying the stereo image stream to a surgeon. The systemalso includes a stereo image to 3D surface extraction module,implemented as machine readable instructions stored in memory of acomputer, that is capable of, when the instructions are executed by aprocessor of the computer, generating a first 3D model of the surgicalfield from the stereo image streams; a registration module, implementedas machine readable instructions stored in the memory, that is capableof, when the instructions are executed by the processor, co-registeringannotating data with the first 3D model; and a stereo image enhancer,implemented as machine readable instructions stored in the memory, thatis capable of, when the instructions are executed by the processor,graphically overlaying at least part of the annotating data onto thestereo image stream to form an enhanced stereo image stream for displayby the stereo viewer, wherein the enhanced stereo stream enhances thesurgeon's perception of the surgical field.

A system designated AA including the system designated A, wherein theregistration module aligns the annotating data with the first 3D model.

A system designated AB including the system designated A or AA furtherincluding a 3D model generator for generating a second 3D model from theannotating data, wherein the registration module co-registers the second3D model with the first 3D model; and an image generator configured togenerate a graphical image of the second 3D model based upon thesurgical field.

A system designated AC including the system designated A, AA or ABwherein the annotating data comprises one of a preoperative, anintraoperative and a postoperative radiological study selected from thegroup consisting of magnetic resonance imaging (MRI), functionalmagnetic resonance imaging (fMRI), x-ray computed tomography (CT),Single-photon emission computed tomography (SPECT), and positronemission tomography (PET).

A system designated AD including the system designated AB or AC, theimage generator configured to generate a cross sectional view of thesecond 3D model based upon an indicated surgical plane within thesurgical field.

A system designated AE including the system designated A, AA, AB, AC, orAD wherein the annotating data further comprises physiological studydata selected from the group including EEG, evoked potentials, andmagnetoencephalography (MEG).

A system designated AF including the system designated A, AA, AB, AC, ADor AE, the registration module further including an alignment refinerconfigured to adjust the registration of the annotating data with the 3Dmodel based upon matching of features within the 3D model and featureswithin the annotating data.

A system designated AF including the system designated A, AA, AB, AC,AD, AE, or AF, the registration module further including a deformationmodeler configured to deform the annotating data based upon a determinedtissue deformation within the surgical field.

A system designated AG including the system designated AF, the tissuedeformation resulting from one or more of skull removal, retraction, andincision.

A system designated AH including the system designated A, AA, AB, AC,AD, AE, AF, or AG and further including a first controllable filtercoupled with a first of the two cameras to filter light entering thefirst of the two cameras from the surgical field; a first controllablelight source for illuminating the surgical field with light of adetermined fluorescent stimulus wavelength; and a fluorescence depthprocessor, implemented as machine readable instructions stored in thememory, that is capable of, when the instructions are executed by theprocessor, generating a 3D fluorescence model; wherein the stereo imageenhancer graphically overlays at least a portion of the 3D fluorescencemodel onto the enhanced stereo image stream.

A system designated AK including the system designated AH, furtherincluding a second controllable filter coupled with a second of the twocameras to filter light entering the second of the two cameras from thesurgical field; and a hyperspectral image processor, implemented asmachine readable instructions stored in the memory, that is capable of,when the instructions are executed by the processor, controlling thefirst and second filters to capture a stereo hyperspectral image streamand generate a 3D hyperspectral model; wherein the stereo image enhancergraphically overlays at least a portion of the 3D hyperspectral modelonto the enhanced stereo image stream.

A system designated AL including the system designated A, AA, AB, AC,AD, AE, AF, or AG, further including a first and a second hyperspectralimaging camera for capturing a stereo hyperspectral image stream of thesurgical field; and a hyperspectral image processor, implemented asmachine readable instructions stored in the memory, that is capable of,when the instructions are executed by the processor, generating a 3Dhyperspectral model from the stereo hyperspectral image stream; whereinthe stereo image enhancer graphically overlays at least a portion of the3D hyperspectral model onto the enhanced stereo image stream.

A system designated B for graphically representing and displaying afirst and second annotating data in relation to a surgical field,including an interactive display device; a registration module,implemented as machine readable instruction stored in a memory of acomputer, that is capable of, when the instructions are executed by aprocessor of the computer, co-registering the first and secondannotating data; and a stereo image enhancer, implemented as machinereadable instruction stored in the memory, that is capable of, when theinstructions are executed by the processor, graphically overlaying atleast part of the annotating data onto the stereo image stream to forman enhanced stereo image stream for display by the stereo viewer,wherein the enhanced stereo stream enhances the surgeon's perception ofthe surgical field.

A system designated BA including the system designated B, furtherconfigured to review a recorded surgical procedure that is co-registeredwith post operatively acquired imaging and/or physiological data.

A method designated C for surgical navigation with stereo vision,including capturing a stereo image pair of a surgical field; generatinga 3D surface model from the stereo image pair; registeringpreoperatively captured annotating data with the 3D surface model;generating an enhanced stereo image based upon the stereo image pair andthe annotating data; wherein the annotating data is graphically overlaidonto the stereo image pair.

A method designated CA including the method designated C, the step ofregistering further including generating a 3D model based upon theannotating data; and registering the 3D model to the 3D surface model.

A method designated CB including the method designated C or CA, furtherincluding adjusting the annotating data based upon matched features ofthe 3D surface model and features of the annotating data.

A method designated CC including the method designated C, CA, or CB, thestep of adjusting comprising one or more of shifting, rotating, warping,and scaling the annotating data.

A method designated CD including the method designated C or CA, the stepof adjusting including generating a mechanical model of the tissuewithin the surgical field to determine the adjustment of the enhanceddata.

A method designated CE including the method designated C, CA, CB, or CC,further including recording the stereo image pair.

A method designated CF including the method designated C, CA, CB, or CC,further including recording the enhanced stereo image pair.

A method designated CG including the method designated C, CA, CB, or CC,wherein the annotating data comprises one of a preoperative, anintraoperative and a postoperative radiological study selected from thegroup consisting of magnetic resonance imaging (MRI), functionalmagnetic resonance imaging (fMRI), x-ray computed tomography (CT),Single-photon emission computed tomography (SPECT), and positronemission tomography (PET).

A method designated CH including the method designated C, CA, CB, CC,CD, CE, CF, or CG, wherein the annotating data further comprises aphysiological study selected from the group including EEG, evokedpotentials, and magnetoencephalography (MEG).

Changes may be made in the above methods and systems without departingfrom the scope hereof. It should thus be noted that the matter containedin the above description or shown in the accompanying drawings should beinterpreted as illustrative and not in a limiting sense. The followingclaims are intended to cover all generic and specific features describedherein, as well as all statements of the scope of the present method andsystem, which, as a matter of language, might be said to falltherebetween.

What is claimed is:
 1. A system for generating a photographic image of asurgical field, comprising: a stereo image capture device for capturinga stereo image stream from two optical cameras imaging the surgicalfield during a surgical procedure; a stereo viewer for displaying thestereo image stream to a surgeon; a stereo image to 3D surface modelextraction module, implemented as machine readable instructions storedin memory of a computer, adapted to, when the instructions are executedby a processor of the computer, generate a first 3D model of thesurgical field from the stereo image stream; a registration module,implemented as machine readable instructions stored in the memory,adapted to, when the instructions are executed by a processor of thecomputer, co-register annotating data with the first 3D model; and astereo image enhancer, implemented as machine readable instructionsstored in the memory, adapted to, when the instructions are executed bya processor of the computer, graphically overlay at least part of theannotating data onto the stereo image stream to form an enhanced stereoimage stream for display by the stereo viewer, wherein the enhancedstereo stream enhances the surgeon's perception of the surgical field.2. The system of claim 1, wherein the registration module aligns theannotating data with the first 3D model.
 3. The system of claim 1,further comprising: a 3D model generator for generating a second 3Dmodel from the annotating data, wherein the registration moduleco-registers the second 3D model with the first 3D model; and an imagegenerator configured to generate a graphical image of the second 3Dmodel based upon the surgical field.
 4. The system of claim 3, whereinthe annotating data comprises one of a preoperative, an intraoperativeand a postoperative radiological study selected from the groupconsisting of magnetic resonance imaging (MRI), functional magneticresonance imaging (fMRI), x-ray computed tomography (CT), Single-photonemission computed tomography (SPECT), and positron emission tomography(PET).
 5. The system of claim 4, the image generator configured togenerate a cross sectional view of the second 3D model based upon anindicated surgical plane within the surgical field.
 6. A system forgenerating a photographic image of a surgical field, comprising: astereo image capture device for capturing a stereo image stream from twocameras imaging the surgical field during a surgical procedure; a stereoviewer for displaying the stereo image stream to a surgeon; a stereoimage to 3D surface extraction module, implemented as machine readableinstructions stored in memory of a computer, configured to, when theinstructions are executed by a processor of the computer, generate afirst 3D model of the surgical field from the stereo image stream; aregistration module, implemented as machine readable instructions storedin the memory, configured to, when the instructions are executed by theprocessor, co-register annotating data with the first 3D model; a stereoimage enhancer, implemented as machine readable instructions stored inthe memory, configured to, when the instructions are executed by theprocessor, graphically overlay at least part of the annotating data ontothe stereo image stream to form an enhanced stereo image stream fordisplay by the stereo viewer, wherein the enhanced stereo streamenhances the surgeon's perception of the surgical field; a 3D modelgenerator for generating a second 3D model from the annotating data,wherein the registration module co-registers the second 3D model withthe first 3D model; and an image generator configured to generate agraphical image of the second 3D model based upon the surgical field;wherein the annotating data further comprises physiological study dataselected from the group including EEG, evoked potentials, andmagnetoencephalography (MEG).
 7. The system of claim 1, the registrationmodule further comprising an alignment refiner configured to adjust theregistration of the annotating data with the 3D model based uponmatching of features within the 3D model and features within theannotating data.
 8. The system of claim 1, the registration modulefurther comprising a deformation modeler configured to deform theannotating data based upon a determined tissue deformation within thesurgical field.
 9. The system of claim 8, the tissue deformationresulting from one or more of skull removal, retraction, and incision.10. The system of claim 4, further comprising: a first controllablefilter coupled with a first of the two cameras to filter light enteringthe first of the two cameras from the surgical field; a firstcontrollable light source for illuminating the surgical field with lightof a determined fluorescent stimulus wavelength; and a fluorescencedepth processor, implemented as machine readable instructions stored inthe memory, adapted to, when the instructions are executed by theprocessor, generate a 3D fluorescence model; wherein the stereo imageenhancer graphically overlays at least a portion of the 3D fluorescencemodel onto the enhanced stereo image stream.
 11. The system of claim 10,further comprising: a second controllable filter coupled with a secondof the two cameras to filter light entering the second of the twocameras from the surgical field; and a hyperspectral image processor,implemented as machine readable instructions stored in the memory,adapted to, when the instructions are executed by the processor, controlthe first and second filters to capture a stereo hyperspectral imagestream and generate a 3D hyperspectral model; wherein the stereo imageenhancer graphically overlays at least a portion of the 3D hyperspectralmodel onto the enhanced stereo image stream.
 12. The system of claim 1,further comprising: a first and a second hyperspectral imaging camerafor capturing a stereo hyperspectral image stream of the surgical field;and a hyperspectral image processor, implemented as machine readableinstructions stored in the memory, adapted to, when the instructions areexecuted by the processor, generate a 3D hyperspectral model from thestereo hyperspectral image stream; wherein the stereo image enhancergraphically overlays at least a portion of the 3D hyperspectral modelonto the enhanced stereo image stream.
 13. The system of claim 3,further configured to review a recorded surgical procedure that isco-registered with post operatively acquired imaging and/orphysiological data.
 14. A method for surgical navigation with stereovision, comprising: capturing a stereo image pair of a surgical fieldduring an operation on the surgical field; generating a 3D surface modelfrom the stereo image pair; registering annotating data with the 3Dsurface model, said registering including adjusting the annotating databased upon matched features of the 3D surface model and features of theannotating data, said adjusting including generating a mechanical modelof the tissue within the surgical field to determine an adjustment;generating an enhanced stereo image based upon the stereo image pair andthe annotating data as registered with the 3D surface model; wherein theannotating data is graphically overlaid onto the stereo image pair asregistered with the 3D surface model.
 15. The method of claim 14, thestep of registering further comprising: generating a 3D model based uponthe annotating data; and registering the 3D model to the 3D surfacemodel.
 16. The method of claim 14, the step of adjusting comprising oneor more of shifting, rotating, warping, and scaling the annotating data.17. The method of claim 20, further comprising recording the stereoimage pair.
 18. The method of claim 20, further comprising recording theenhanced stereo image pair.
 19. The method of claim 14, wherein theannotating data comprises one of a preoperative, an intraoperative and apostoperative radiological study selected from the group consisting ofmagnetic resonance imaging (MRI), functional magnetic resonance imaging(fMRI), x-ray computed tomography (CT), Single-photon emission computedtomography (SPECT), and positron emission tomography (PET).
 20. A methodfor surgical navigation with stereo vision, comprising: capturing astereo image pair of a surgical field during an operation on thesurgical field; generating a 3D surface model from the stereo imagepair; registering annotating data with the 3D surface model; generatingan enhanced stereo image based upon the stereo image pair and theannotating data; wherein the annotating data is graphically overlaidonto the stereo image pair, and wherein the annotating data furthercomprises a physiological study selected from the group including EEG,evoked potentials, and magnetoencephalography (MEG).
 21. The method forsurgical navigation of claim 20 wherein the annotating data is derivedfrom magnetic resonance imaging (MRI) or computed tomography (CT) imagestacks.
 22. The method for surgical navigation of claim 20 wherein theannotating data comprises physiological data.
 23. The system of claim 1wherein the annotating data comprises physiological data.